Constant-Time Sliding Window Aggregation

Sliding-window aggregation is a widely-used approach for extracting insights from the most recent portion of a data stream. Most aggregation operations of interest can be cast as binary operators that are associative, but not necessarily commutative nor invertible. However, non-invertible operators are nontrivial to support efficiently. The best existing algorithms for this setting require Oplog nq aggregation steps per window operation, where n is the window size at that point. This paper presents DABA, a novel algorithm that significantly improves upon this time bound, assuming the sliding window has FIFO semantics. DABA requires only Op1q aggregation steps, in the worst case, per window operation. As such, DABA asymptotically improves the performance of sliding-window aggregation without restricting the operator to be invertible. Furthermore, our experimental results demonstrate that these theoretic improvements hold in practice. DABA is a significant improvement over the state-of-the-art for both throughput and latency.